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Abstract 

Collaborative problem solving and collaborative skills are considered necessary skills for success 
in today's world. Collaborative problem solving is defined as problem solving activities that involve 
interactions among a group of individuals. Large-scale and small-scale assessment programs 
increasingly use collaborative group tasks in which students work together to solve problems or to 
accomplish projects. 

This study attempts to research the role of feedback on a computer-based collaborative problem 
solving task by extending Hsieh and O'Neil's (2002) computer-based collaborative knowledge mapping 
study. In their study, groups of students searched a web environment of information to improve a 
knowledge map. Various types of feedback were investigated. They found that searching has a negative 
relationship with group outcome (knowledge map scores). By teaching searching and by providing 
different types of feedback, this study explores the effects of students' teamwork and problem solving 
processes on students' knowledge mapping performance. Moreover, the effects of two types of feedback 
(adapted knowledge of response feedback and task-specific adapted knowledge of response feedback) 
were also investigated. 

One hundred and twenty college students (60 groups) participated in the main study. The 
students were randomly assigned either to be a group leader whose responsibility was to construct the 
map or to be a group searcher whose responsibility was to help the leader construct the map by seeking 
information and accessing feedback from the Web environment. Results showed that task-specific 
adapted knowledge of response feedback was significantly more beneficial to group outcome than 
adapted knowledge of response feedback. In addition, as predicted, for the problem solving process, 
information seeking including request of feedback, browsing, searching for information and searching 
using Boolean operators were all significantly related to group outcome for both groups. 
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Lastly, this study confirmed that computer-based performance assessment in collaborative 
problem solving was effective. The collaboration between team members and individual students' 
problem solving processes and strategies were effectively recorded by the computers. In addition, the use 
of computers to assess and report group interaction and students' thinking processes was proven to be 
more inexpensive and less time consuming than other alternatives. 



Introduction 

Many research studies on collaborative problem solving have shown a positive 
effect on students' cognitive improvement (e.g.. Arts, Gijselaers, & Segers, 2002). 
Recently, computer networks and simulations have been used and proven effective as 
an assessment tool to measure collaboration skills and problem-solving skills (e.g., 
Hsieh & O'Neil, 2002). Another influence brought about by advances in computer 
technology is the new emphasis on the skills needed for electronic information seeking 
and processing (Covington, 1998). However, many students and teachers alike still lack 
basic information technology knowledge and skills (Smith & Broom, 2003). Current 
curriculum, instruction, and assessment do not adequately make use of the capabilities 
of today's networked information systems. Research has shown that expert searchers 
locate information faster and use more search logic, such as Boolean operators, in their 
queries than novice searchers (Lazonder, 2000). Thus research has pointed to a need for 
training on Boolean search strategies, and a few studies have also demonstrated its 
effectiveness. However, in Hsieh and O'Neil's study, searching was unexpectedly 
negatively related to performance on a team knowledge mapping task. 

By combining Hsieh and O'Neil's (2002) methodological approach with 
instruction in search strategies, this study investigated student collaborative problem- 
solving and team processes on a computer-based knowledge mapping team task with 
special attention to the effectiveness of feedback. In Hsieh and O'Neil's study, two 
levels of feedback, adapted feedback and knowledge of response feedback, were 
compared. Knowledge of response feedback is feedback that informs the learner 
whether the answer is correct. Adapted feedback was designed in computer-based 
instruction with attention to customizing feedback for the user's needs rather than 
providing just one fixed form of feedback. They demonstrated that adapted feedback 
teams outperformed knowledge of response feedback teams. Continuing in the same 
fashion, the current study compared two levels of adapted feedback — adapted feedback 
and task-specific adapted feedback. 
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The measurement of teamwork processes and problem-solving processes in this 
study was based on the model developed by the National Center for Research on 
Evaluation, Standards, and Student Testing (CRESST). The teamwork model consists of 
six teamwork skills: "(a) adaptability, (b) coordination, (c) decision making, 

(d) interpersonal, (e) leadership, and (f) communication (O'Neil, Chung, & Brown, 1997, 
p. 413). The problem-solving model consists of three sub-elements: (a) content 
understanding, (b) problem-solving strategies, and (c) self-regulation. 

In the current study, content understanding was measured by a knowledge map 
created with and recorded by computer software. A knowledge map is a graphical 
representation consisting of nodes and links. Nodes represent terms (standing for a 
concept) in the domain of knowledge. Links represent the relationships between nodes. 
A proposition is the combination of two nodes joined by a link. 

Hsieh and O'Neil (2002) used a simulated Internet Web to evaluate student 
collaborative problem-solving strategies and outcomes in a computer-based knowledge 
mapping environment. They successfully showed that adapted feedback was more 
effective in improving students' performance than knowledge of response feedback. 
The current study used the same simulated Internet Web space, in which students 
searched to find information to improve their knowledge maps, and the same 
simulation also provided students with feedback on their maps. 

In the literature, feedback was regarded as an important resource to assist the 
learning process. Hsieh and O'Neil (2002) demonstrated that feedback access was 
positively related to outcome performance. However, even though their feedback 
provided participants with a direction as to "what" area to improve on their knowledge 
map, it did not provide practical tips on "how" to improve the map. The present study 
argued that if search tips were provided in the feedback, students would become more 
effective and efficient in locating the information and in turn improve the overall result 
of their map. Therefore, we modified Hsieh and O'Neil's original task by providing 
examples of how to use Boolean operators in addition to information on the area(s) into 
which participants should put more effort. 

Because feedback with search tips was provided, participants in the current 
study were expected to perform better than Hsieh and O'Neil's (2002) participants in 
general. In addition, increased use of Boolean operators when doing the simulated Web 
search was expected. 

There were four hypotheses in this study. 
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Hypothesis 1: Students will perform better on the environmental science 
knowledge mapping task if they receive task-specific adapted feedback than if they 
receive adapted feedback. 

Hypothesis 2: Students who receive task-specific adapted feedback will be more 
likely to use decision-making and leadership team processes than students who receive 
adapted feedback. 

Hypothesis 3: Information seeking will have positive effects on team outcome in 
environmental science knowledge mapping. 

3.1 Browsing and searching will be positively related to team outcome. 

3.2 The number of requests for feedback will be positively related to team 
outcome. 

Hypothesis 4: Searching using Boolean operators will have positive effects on 
students' problem-solving strategies and team outcome on the knowledge mapping 
task. 

4.1 The more frequently Boolean operators are used, the higher the map score 
on the knowledge map. 

4.2 The task-specific adapted feedback team will use Boolean operators more 
frequently in their searching than the adapted feedback team. 

Method 



Participants 

Participates were 120 college students (60 dyads), 18 years of age or older. 

Networked Knowledge Mapping System 

The mapping system was based on the networked knowledge mapping system 
developed by Schacter, Herl, Chung, Dennis, and O'Neil (1999) and furthered modified 
by Hsieh and O'Neil (2002). Using this system, teams of two participants (dyads) 
created a knowledge map on environmental science by exchanging messages in a 
collaborative environment and by searching for relevant information in a simulated 
World Wide Web environment. The participants in a dyad were randomly assigned to 
the role of leader or searcher. The leader was solely responsible for creating the 
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knowledge map, and the searcher was solely responsible for accessing the simulated 
World Wide Web to find information and to request feedback. 

The knowledge map concepts were 18 predefined important concepts identified 
by content experts: atmosphere, bacteria, carbon dioxide, climate, consumer, 
decomposition, evaporation, food chain, greenhouse gases, nutrients, oceans, oxygen, 
photosynthesis, producer, respiration, sunlight, waste, and water cycle. The links for the 
knowledge map were also predefined by content experts: CAUSES, INFLUENCES, 
PART OF, PRODUCES, REQUIRES, USED FOR, and USES. There were altogether 37 
predefined messages with 37 corresponding buttons. When a participant clicked on a 
button, the corresponding message was sent instantly to both team members' 
computers. The simulated World Wide Web environment contained more than 200 Web 
pages with approximately 500 images and diagrams about environmental science. 

Feedback 

The searcher was allowed to access feedback to find out how well his or her team 
was performing. Feedback was provided in one of two categories: adapted feedback 
and task-specific adapted feedback; both were based on comparing students' 
knowledge map performance to that of experts'. Adapted feedback in the present study 
was identical with Hsieh and O'Neil's (2002) feedback condition. It pointed out 
concepts that needed improvement. For example, as can be seen in Figure 1, 
"atmosphere" needed a lot of improvement according to the feedback. 
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Your map has been scored against an expert's map in environmental science. The feedback tells 
you: 

• How much you need to improve each concept in your map (i.e., A lot. Some, 

A little) 

Use this feedback to help you search to improve your map. 



A Lot 


Some 


A Little 


atmosphere. 


food chain. 


photosynthesis. 


carbon dioxide. 


decomposition. 


oxygen. 


respiration. 


consumer. 


waste. 


evaporation, 

sunlight, 

water cycle, 

oceans, 

bacteria 

greenhouse 

gases 


producer 


climate 



Improvement: You have improved the "food chain" from needing "A lot of improvement" to 
"Some improvement" category. 

Strategy: It is most useful to investigate information for the "A lot" and "Some" categories rather 
than the "A little" category. For example, improving "atmosphere" or "climate" first rather than 
"evaporation." 



Figure 1. Example of adapted feedback in Hsieh and O'Neil's (2002) study. 



Task-specific adapted feedback, used in the current study, included the 
information contained in adapted feedback and Boolean search strategy tips. Figure 2 
shows an example of feedback provided in the present study. Search tips were adapted 
from the Firstsearch help guide at http://newfirstsearch.oclc.org/ 

As seen in Figure 2, the feedback provided participants a direction as to what 
area to improve for search and task performance, and also how to improve the 
performance. It should be noted that the search tips were task-specific adaptive 
knowledge of results response feedback to help teams to improve their searches. 
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Your map has been scored against an expert's map in environmental science. The feedback tells you: 

How much you need to improve each concept in your map (i.e., A lot. Some, A little). 

Use this feedback to help you search to improve your map 



A Lot 


Some 


A Little 


atmosphere. 


food chain. 


photosynthesis. 


carbon dioxide. 


decomposition. 


oxygen. 


respiration. 


consumer. 


waste. 


evaporation, 

sunlight, 

water cycle, 

oceans, 

bacteria 

greenhouse 

gases 


producer 


climate 



Improvement: You have improved the "food chain" from needing "A lot of improvement" to "Some 
improvement" category. 

General Strategy: It is most useful to investigate information for the "A lot" and "Some" categories 
rather than the "A little" category. For example, improving "atmosphere" or "climate" first rather than 
"evaporation". 

Search strategy 1: Use the Boolean operators AND to combine search terms when you need to expand or 
narrow a search. AND retrieves only records that contain all search terms. Use this operator to narrow or 
limit a search. 

oxygen AND atmosphere 




If you type: oxygen AND atmosphere 
It searches for: Only records containing both oxygen and atmosphere 

If you type: oxygen AND atmosphere AND carbon dioxide 
It searches for: Only records containing all three search terms— oxygen, and atmosphere, and carbon 
dioxide 

Search strategy 2: If using the Boolean operator AND ends up with no relevant pages found. Use the 
Boolean operator OR to retrieve all records that contain one or both of the search terms. Use this operator 
to expand a search. 

oxygen OR atmosphere 




If you type: Oxygen OR atmosphere 

It searches for: Records containing oxygen, records containing atmosphere, and records containing both 
Figure 2. Example of task-specific adapted feedback in the present study. 
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Measures 



Team Outcome Measure 

Team outcome measures were computed by comparing the semantic content 
score of a team's knowledge map to that of a set of four experts' maps (Schacter et al., 
1999). The following description shows how these outcomes were scored. First, the 
semantic score was based on the semantic propositions in experts' knowledge maps and 
was calculated by categorized map scoring (Herl, O'Neil, Chung, & Schacter, 1999). 
Using this method, all seven links were categorized into four classifications. CAUSES 
and INFLUENCES were classified as the "casual" category and marked as String 1. 
REQUIRES, USED FOR, and USES were classified as the "conditional" category and 
marked as String 2. PART OF and PRODUCES, the remaining two links, were classified 
individually and marked as Strings 3 and 4 respectively. Every proposition in a student 
map was compared against each proposition in the four experts' maps using Strings 1- 
4. One match was scored as one point. The average score across all four experts was the 
semantic score of the map. For example, if a student team made a proposition such as 
"Oceans PART OF Water cycle," this proposition would be first categorized into 
"Oceans 3 Water cycle" and then compared with the four experts' propositions. A score 
of 1 means this proposition was the same as a proposition in the map of an expert. A 
score of zero means this proposition was not the same as any of the experts' 
propositions. 

Teamwork Process Measures 

Teamwork process scores were calculated by adding the number of messages 
both members in a team sent from each teamwork process category. That is, if a team 
leader sent 7 messages from the adaptability category, then that person's individual- 
level adaptability score was 7. If both the leader and the searcher in a team each sent 7 
messages from the adaptability category, the team-level adaptability score was 14. 

Information Seeking and Feedback Behavior Measures 

Information seeking and feedback behavior were measured by three activities: 
browsing, searching, and requesting feedback. Browsing was measured by how many 
times a searcher selected Web pages or clicked on any hypertext within the Web pages. 
Each time a searcher selected a page or clicked on any hypertext, a point was added. For 
searching, one point was awarded for simple searches. For example, when a searcher 
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typed "oxygen" as the search string, one point was awarded. An additional point was 
awarded if the search used Boolean search strategies. For example, "oxygen AND 
sunlight" would be counted as two points: one for a simple search and one for using the 
Boolean operator AND. The score for feedback request was calculated as the number of 
times the team requested feedback. 



Results 



Teamwork Process Measure 

Table 1 presents teamwork process measure frequency counts for all six 
teamwork processes. In general, the reliability of the teamwork process measure was 
unacceptably low. Alpha reliability ranged from -0.03 (leadership) to 0.48 
(interpersonal). Examination of the "Alpha if item deleted" data indicated that deletion 
of some messages would improve the reliability. After deleting items, the final alpha 
reliability for these processes ranged from 0.29 for decision making to 0.65 for 
coordination. Alpha reliability for the other subscales ranged from 0.37 for adaptability, 
0.29 for decision making, 0.46 for leadership and 0.48 for communication. 

The frequency counts were calculated by adding the number of usage for the 
individual messages in each teamwork process, after item deletion, and then dividing 
by the number of teams. In addition, because some team processes had more messages 
than others, a new metric was created by using a mean. The means in Table 1 were 
calculated by taking the mean of the frequency count for a category and dividing it by 
the number of messages left in that category after item deletion. Using adaptability as 
an example, the mean of 4.80 was divided by 3, the number of messages left after item 
deletion; therefore, the final mean was calculated to be 1.60. 
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Table 1 



Frequency Count of Teamwork Processes: Team Level ( N = 60) 



Teamwork process 


No. of 
messages 


M 


SD 


Min. 


Max. 


Adaptability 


3 


1.60 


3.26 


.00 


18.00 


Coordination 


2 


0.50 


1.71 


.00 


8.00 


Decision making 


4 


2.01 


7.98 


.00 


39.00 


Interpersonal 


5 


1.39 


5.32 


.00 


23.00 


Leadership 


2 


0.73 


1.78 


.00 


7.00 


Communication 


2 


0.79 


2.22 


.00 


10.00 



Table 2 shows the intercorrelations for the teamwork process variables and the 
two feedback condition teams. For the adapted feedback team, six significant 
correlations were found. First, decision making was significantly correlated with 
adaptability (r = 0.39, p < 0.05). In addition, a significant correlation was found between 
communication and leadership (r = 0.76, p < 0.01). Moreover, "all messages sent" 
significantly correlated with adaptability (r = 047, p < 0.01), decision making (r = 0.80, p 
< 0.01), and interpersonal (r = 0.71, p < 0.01). Since "all messages sent" was calculated as 
the sum of all messages in the six individual categories, these results were expected. 
Last, team outcome was negatively correlated with leadership (r = -0.39, p < 0.05). This 
correlation indicates that the greater the number of leadership messages sent, the worse 
the team outcome was. This result was unexpected. 

For the task-specific adapted feedback team, 10 significant correlations were 
found. First, adaptability was significantly correlated with leadership (r = 0.37, p < 0.05) 
and communication (r = 0.52, p < 0.01). Second, coordination was significantly 
correlated with interpersonal (r = 0.60, p < 0.01). Leadership, in addition to being 
significantly correlated with adaptability, as mentioned above, was also significantly 
correlated with communication (r = 0.43, p < 0.05). These significant relationships 
indicate that an increase in the number of leadership messages sent resulted in an 
increase in the number of adaptability messages and communication messages sent, or 
vice versa, by the task-specific adapted feedback team. 
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In addition, "all messages sent" was significantly correlated with adaptability (r 
= 0.43, p < 0.05), decision making (r = 0.59, p < 0.01), and interpersonal (r = 0.47, p < 0.01) 
processes. Communication (r = 0.39, p < 0.05) was significantly correlated with "all 
messages sent." Last, team outcome was significantly correlated with interpersonal 
messages sent and with "all messages sent." This set of correlations indicates that the 
greater the number of interpersonal messages sent, the better the team outcome. 

Table 2 

Intercorrelations for Team-Level Teamwork Process Measures and Team Outcome for the 

Adapted Feedback Team ( n = 30) and for the Task-Specific Adapted Feedback Team ( n = 30) 





1 


2 


3 


4 


5 


6 


7 


8 


1. Adaptability 




0.31 


-0.24 


0.16 


0.37* 


0.52** 


0.43* 


0.16 


2. Coordination 


0.10 




-0.34 


0.60** 


-0.18 


-0.15 


0.22 


0.21 


3. Decision 


0.39* 


0.14 




-0.18 


-0.11 


0.00 


0.59** 


0.12 


making 


4. Interpersonal 


0.17 


0.32 


0.34 




-0.16 


-0.04 


0.47** 


0.49** 


5. Leadership 


-0.19 


0.03 


-0.04 


0.02 




0.43* 


0.13 


0.18 


6. Communi- 


-0.09 


-0.12 


-0.01 


-0.02 


0.76** 




0.39* 


0.15 


cation 


7. All messages 


0.47** 


0.35* 


0.80** 


0.71** 


0.27 


0.27 




0.48** 


sent 


8. Outcome 


0.27 


0.08 


-0.05 


0.33 


-0.39* 


-0.31 


0.06 




performance 



Note. Inter cor relations for the adapted feedback team are shown below the diagonal. 
*p < .05, two-tailed. **p < .01, two-tailed. 
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Team Outcome Measure 

The mean team outcome for the adapted feedback team was 10.96, and for the 
task-specific adapted feedback team it was 12.96. The mean difference for team outcome 
for the two feedback treatment teams was statistically significant, f (29) = 2.09, p = 0.046. 

The maximum possible team outcome score was 23.31. This score was calculated 
as the average of the four expert map scores. Thus the students in this study 
demonstrated 51% of the experts' knowledge, whereas the students in Hsieh and 
O'Neil's (2002) study demonstrated 42% of the experts' knowledge. The mean 
difference for team outcome in this study and Hsieh and O'Neil's study was statistically 
significant, f (59) = 3.75, p = 0.00. In Hsieh and O'Neil's study, the participants were high 
school students, whereas in this study, the participants were college students. 

Problem-Solving Process Measures 

Table 3 presents descriptive statistics for problem-solving variables in this study. 
As can be seen in Table 3, for the total team ( N = 60), the mean score for content 
understanding (knowledge map score) was 10.96. For problem-solving strategies, for 
the total team, the mean for "browsing" was 88.47, the mean for "searching" was 16.17, 
the mean for "Boolean operators used" was 4.00, and the mean for feedback was 19.53. 
A two-tailed t test shows that a significant difference between teams was found for 
browsing, f (29) = 2.52, p = .02, and numbers of Boolean operator used, f (29) =2.52, p = 
0.02, both for the task-specific adapted feedback team. No significant difference was 
found for searching, t( 29) = 0.57, p = 0.58, or feedback accessing, t( 29) = 0.24, p = 0.82. 
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Table 3 



Descriptive Statistics for Problem-Solving Variables 





Total 
(N =60) 


Adapted feedback 
team (n = 30) 


Task-specific 
adapted feedback 
team (n = 30) 


Information seeking 
and feedback behavior 


M 


SD 


M 


SD 


M 


SD 


Content 

understanding 














Knowledge map 
score 

Problem-solving 

strategies 


11.97 


3.67 


10.96 


3.26 


12.96 


3.84 


Browsing 


113.31 


73.79 


88.47 


54.53 


138.17 


82.62 


Searching 


17.15 


12.84 


16.17 


12.61 


18.13 


13.21 


Boolean operators 
used 


5.62 


5.06 


4.00 


3.90 


7.23 


5.62 


Feedback 

Self-regulation 


20.01 


15.07 


19.53 


15.38 


20.50 


15.01 


Planning (8 items) 


3.14 


0.44 


3.19 


0.45 


3.08 


0.42 


Self-checking (8 
items) 


3.03 


0.50 


3.11 


0.52 


2.93 


0.46 


Effort (8 items) 


3.09 


0.47 


3.17 


0.49 


3.01 


0.45 


Self-efficacy (8 
items) 


2.93 


0.60 


3.04 


0.61 


2.80 


0.58 



There were many significant correlations found among the problem-solving 
process measures. As may be seen in Table 4, for the total team (N = 60), knowledge 
map score (content understanding) was significantly related to browsing (r = 0.40, 
p <0.01), searching (r = 0.43, p < 0.01), Boolean operators used (r = 0.42, p < 0.01), and 
feedback accessing (r = 0.38, p < 001). Furthermore, the four problem-solving strategies 
(browsing, searching. Boolean operators used, and feedback accessing) were correlated 
to one another significantly. Browsing was significantly related with searching (r = 0.72, 
p < 0.01), Boolean operators used (r = 0.75, p < 0.01), and feedback accessing (r = 0.74, p 
< 0.01). Searching was significantly related to Boolean operators used (r = 0.92, p < 0.01) 
and feedback accessing (r = 0.84, p < 0.01). Boolean operators used was significantly 
related to feedback accessing (r = 0.79, p < 0.01). As for the four self-regulation measures 
(planning, self-checking, effort, and self-efficacy), they were all significantly related to 
one another. However, no significant correlation was found among the four self- 
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regulation measures (Hong, O'Neil, & Feldon, in press; O'Neil & Herl, 1998) and the 
three problem-solving strategies. In addition, there was no significant correlation found 
among the four self-regulation measures and the team outcome. 

Table 4 



Correlations of Team Outcome and Problem-Solving Variables 







Team outcome— knowledge map score 






Adapted knowledge 


Task-specific adapted 






of response 


knowledge of response 


Problem-solving variables 


Total 


feedback team 


feedback team 




(N = 60) 


(n = 30) 


(n = 30) 


Problem-solving 








strategies 








Browsing 


0.40** 


0.36** 


0.36** 


Searching 


0.43** 


0.44** 


0.40** 


Boolean operators 
used 


0.42** 


0.41** 


0.37** 


Feedback 


0.38** 


0.35** 


0.42** 


Self-regulation 








Planning 


-0.11 


-0.08 


-0.07 


Self-checking 


0.05 


0.04 


0.18 


Effort 


-0.06 


-0.04 


0.00 


Self-efficacy 


-0.12 


-0.08 


-0.07 



*p < .05, two-tailed. **p < .01, two-tailed. 
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For the adapted feedback team, knowledge map score (content understanding) 
was significantly related to browsing (r = 0.36, p < 0.01), searching (r = 0.44, p < 0.01), 
Boolean operators used (r = 0.41, p < 001), and feedback accessing (r = 035, p < 0.01). 
There was no significant relationship found between knowledge map score and self- 
regulation measures (planning, self-checking, effort, and self-efficacy). Flowever, each 
of the four self-regulation measures was significantly correlated with the other three, 
and each of the four problem-solving strategies (browsing, searching. Boolean operators 
used, and feedback accessing) was significantly correlated with the other three. 
Nevertheless, none of the self-regulation measures were correlated significantly with 
any of the problem-solving strategies (browsing, searching. Boolean operators used, 
and feedback accessing). 

The significant correlations found for the task-specific adapted feedback team 
were similar to those found for the adapted feedback team. For the task-specific 
adapted feedback team, knowledge map score (content understanding) was also 
significantly related to browsing (r = 0.36, p < 0.01), searching (r = 0.40, p < 0.01), 
Boolean operators used (r = 0.37, p < 0.01), and feedback accessing (r = 0.42, p < 0.01). 
No significant relationship was found between knowledge map score and self- 
regulation measures (planning, self-checking, effort, and self-efficacy). Each of the four 
self-regulation measures significantly correlated with the other three, and each of the 
four problem-solving strategies significantly correlated with the other three. 
Nevertheless, none of the self-regulation measures was correlated significantly with any 
of the problem-solving strategies (browsing, searching. Boolean operators used, and 
feedback accessing). 

In FIsieh and O'Neil's (2002) study, for the adapted feedback team, searching 
was unexpectedly negatively related to team outcome (r = -0.40, p < 0.05). This was not 
the case in the current study. All four problem-solving strategies were significantly 
positively related to team outcome (knowledge map score) in both feedback conditions. 
One difference between the two studies is that searching was explicitly taught in the 
current study but was not taught in the Hsieh and O'Neil study. 

Tests of Hypotheses 

For Hypothesis 1, the results indicate that the task-specific adapted feedback 
team performed significantly better on the knowledge map than the adapted feedback 
team. A t test (two-tailed) on team outcome between the adapted feedback team and the 
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task-specific adapted feedback team showed a significant difference between the means, 
t( 29) = 2.09, p = 0.046. As expected, students who received task-specific adapted 
feedback (M = 12.96) performed better than those who received adapted feedback only 
(M = 10.96). Thus Hypothesis 1 was supported. 

For Hypothesis 2, the results indicate that students receiving task-specific 
adapted feedback used approximately the same number of decision-making and 
leadership messages in their communications as students receiving adapted feedback. 
The difference between the two teams was not significant for the decision-making team 
process, f (29) = 0.31, p = 0.76, or the leadership team process, f (29) = -1.51, p = 0.14. Thus, 
Hypothesis 2 was not supported. 

For Hypothesis 3, Table 4 shows the correlations between information-seeking 
variables and team outcome. As expected, browsing was significantly related to team 
outcome (knowledge map score). In addition, feedback accessing was positively related 
to team outcome. In other words, the more browsing students did, the higher their team 
outcome (map score) was. Furthermore, the more feedback students requested, the 
better their team outcome (map score) was. Thus Hypotheses 3.1 and 3.2 were 
supported. 

For Hypothesis 4, as may be seen in Table 4, significant positive correlations 
were found for the total team ( N = 60), r = 0.42, p < 0.01, the adapted feedback team, r = 
.41, p < 0.01, and the task-specific adapted feedback team, r = 0.37, p < 0.01. Thus, 
Hypothesis 4.1 was supported. 

Also, as shown in Table 3, students in the task-specific adapted feedback team 
used more Boolean operators than students in the adapted feedback team. A t test (two- 
tailed) showed the mean difference in number of Boolean operators used between the 
two feedback teams to be significant, f (29) = 2.52, p = 0.02. Thus Hypothesis 4.2 was 
supported. 



Discussion 

First, results from the study provide evidence indicating that students who 
received task-specific adapted feedback performed significantly better than students 
who received adapted feedback. This finding was consistent with several themes in the 
feedback literature. For example, adapted feedback was suggested by Sales (1993) to 
customize along one or more dimensions to compensate for the weakness of generic 
feedback that lacks capacity to communicate with learners. Hsieh and O'Neil's (2002) 
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study showed that such adapted feedback was better in assisting students in knowledge 
map construction than knowledge of response feedback. Moreover, Ross and Morrison 
(1993) defined task-specific feedback as elaboration feedback that concentrated on the 
current test item. They found task-specific feedback was better than general feedback in 
assisting learning. Other than Ross and Morrison's study and the current study, there 
has been no research that investigated task-specific adapted feedback. Our results 
support the efficacy of such task-specific adapted feedback. 

Additionally, in regard to feedback presentation style, Kalyuga, Chandler, and 
Sweller (2000) found that low-knowledge or inexperienced learners benefited most from 
feedback combining diagrams with additional text-based information. Foster and 
Macan (2002) showed that providing information about a process or strategy could 
optimize learning during practice. In their study, participants who received this type of 
feedback (called attentional advice) had significantly higher achievement than those 
receiving no advice at all. 

Finally, the results are consistent with a theoretical framework of dynamic 
testing. According to Grigorenko and Sternberg (1998), "Dynamic testing is a collection 
of testing designed to quantify not only the products or even the processes of learning 
but also the potential to learn" (p. 75). In other words, dynamic testing not only 
provides students with accuracy reports on how they are doing on a test, but also 
provides them with information on how to improve the quality of their work on the test 
during the time of the testing. Dynamic testing can do so because it assigns a different 
role to feedback than the one assigned in traditional static testing. In traditional static 
testing, feedback about performance is usually not given during the test. In dynamic 
testing, feedback is given during the test to help assess learning. In this study, the 
students were considered to be low-prior-knowledge students in environmental 
sciences. Combining the research findings on feedback and dynamic testing, two types 
of feedback were created. Adapted feedback was presented in a graphical format with 
extra text based on general task improvement strategies. Task-specific adapted feedback 
was presented in a graphical format with extra text based on general task improvement 
and task-specific strategies. In this study, the feedback was given not after the test but 
during the test to help students learn. As predicted, the task-specific adapted feedback 
team performed significantly better than the adapted feedback team. Students who 
received feedback with task-specific information constructed better knowledge maps 
than students with no task-specific information. The claim made by dynamic testing, 
about the role of feedback as a learning strategy to assist students' performance during 
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the task, was shown to be feasible and successful. Students with task-specific 
information showed a higher use of the Boolean search strategy than students without 
that information. 

The finding that indicated information searching and use of Boolean search 
operators was beneficial for team outcome was in line with the search literature. Baker 
and O'Neil (2003) included use of search engines as one of the requirements of 
technological fluency. In addition, they listed use of Boolean operators as an example of 
a cognitive strategy for use in a problem-solving search task. In an observational study 
of novice users searching information on the World Wide Web, Lazonder (2000) 
suggested instructing students in search logic such as use of Boolean operators to 
improve the quality of their searching. The results from the current study support the 
literature on searching with Boolean operators and training in their use. 

In this study, both collaboration between team members and problem-solving 
processes were effectively measured, administered, scored, and reported by computer. 
However, the results from this study should not be generalized due to the following 
limitations. Firstly, the findings should not be applied to other types of teams due to 
the fact that a dyad is different from other types of teams and team size is an important 
variable in the teamwork literature. Secondly, the college students assessed in this 
study were a very restricted population. Lastly, future study can improve by providing 
stronger evidence on team performance using multiple indicators rather than one single 
indicator as used in this study. 
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